Skip to content

Migrate Gentoo importer to advisory V2#2090

Open
ziadhany wants to merge 8 commits intoaboutcode-org:mainfrom
ziadhany:gentoo-migration
Open

Migrate Gentoo importer to advisory V2#2090
ziadhany wants to merge 8 commits intoaboutcode-org:mainfrom
ziadhany:gentoo-migration

Conversation

@ziadhany
Copy link
Collaborator

@ziadhany ziadhany commented Dec 30, 2025

@ziadhany ziadhany changed the title Add initial migration to Gentoo importer v2 Migrate Gentoo importer to advisory V2 Dec 30, 2025
@ziadhany ziadhany marked this pull request as ready for review January 1, 2026 13:58
@ziadhany
Copy link
Collaborator Author

ziadhany commented Jan 1, 2026

gentoo importer V2 logs:

/home/ziad-hany/PycharmProjects/vulnerablecode/venv/bin/python /home/ziad-hany/PycharmProjects/vulnerablecode/manage.py import localhost:8000 --all 
INFO 2026-02-02 17:54:38.166692 UTC Pipeline [GentooImporterPipeline] starting
INFO 2026-02-02 17:54:38.166802 UTC Step [clone] starting
INFO 2026-02-02 17:54:38.166832 UTC Cloning `git+https://anongit.gentoo.org/git/data/glsa.git`
Importing data using gentoo_importer_v2
INFO 2026-02-02 17:54:43.419509 UTC Step [clone] completed in 5 seconds
INFO 2026-02-02 17:54:43.419620 UTC Step [collect_and_store_advisories] starting
INFO 2026-02-02 17:54:43.447538 UTC Collecting 3,814 advisories
INFO 2026-02-02 17:54:47.695981 UTC Progress: 10% (382/3814) ETA: 38 seconds
INFO 2026-02-02 17:54:51.021923 UTC InvalidVersion constraints version: 1.3* error:'1.3*' is not a valid <class 'univers.versions.GentooVersion'>
INFO 2026-02-02 17:54:51.826717 UTC Progress: 20% (763/3814) ETA: 34 seconds
INFO 2026-02-02 17:54:56.169353 UTC Progress: 30% (1145/3814) ETA: 30 seconds
INFO 2026-02-02 17:54:58.417666 UTC InvalidVersion constraints version: 3.24.48:3 error:'3.24.48:3' is not a valid <class 'univers.versions.GentooVersion'>
INFO 2026-02-02 17:54:58.417815 UTC InvalidVersion constraints version: 3.24.48:3 error:'3.24.48:3' is not a valid <class 'univers.versions.GentooVersion'>
INFO 2026-02-02 17:55:00.338608 UTC Progress: 40% (1526/3814) ETA: 25 seconds
INFO 2026-02-02 17:55:04.620972 UTC Progress: 50% (1907/3814) ETA: 21 seconds
INFO 2026-02-02 17:55:06.363401 UTC InvalidVersion constraints version: 7.4* error:'7.4*' is not a valid <class 'univers.versions.GentooVersion'>
INFO 2026-02-02 17:55:06.363539 UTC InvalidVersion constraints version: 7.3* error:'7.3*' is not a valid <class 'univers.versions.GentooVersion'>
INFO 2026-02-02 17:55:08.923560 UTC Progress: 60% (2289/3814) ETA: 17 seconds
INFO 2026-02-02 17:55:09.268841 UTC InvalidVersion constraints version: 7.4* error:'7.4*' is not a valid <class 'univers.versions.GentooVersion'>
INFO 2026-02-02 17:55:11.946939 UTC InvalidVersion constraints version: 6.9.3:6 error:'6.9.3:6' is not a valid <class 'univers.versions.GentooVersion'>
INFO 2026-02-02 17:55:11.947075 UTC InvalidVersion constraints version: 6.9.3:6 error:'6.9.3:6' is not a valid <class 'univers.versions.GentooVersion'>
INFO 2026-02-02 17:55:12.803704 UTC InvalidVersion constraints version: 7.4* error:'7.4*' is not a valid <class 'univers.versions.GentooVersion'>
INFO 2026-02-02 17:55:12.803886 UTC InvalidVersion constraints version: 7.3* error:'7.3*' is not a valid <class 'univers.versions.GentooVersion'>
INFO 2026-02-02 17:55:13.023839 UTC Progress: 70% (2670/3814) ETA: 13 seconds
INFO 2026-02-02 17:55:14.632720 UTC InvalidVersion constraints version: 7.4* error:'7.4*' is not a valid <class 'univers.versions.GentooVersion'>
INFO 2026-02-02 17:55:14.632854 UTC InvalidVersion constraints version: 7.3* error:'7.3*' is not a valid <class 'univers.versions.GentooVersion'>
INFO 2026-02-02 17:55:16.977535 UTC Progress: 80% (3052/3814) ETA: 8 seconds
INFO 2026-02-02 17:55:21.247673 UTC Progress: 90% (3433/3814) ETA: 4 seconds
INFO 2026-02-02 17:55:25.279534 UTC Progress: 100% (3814/3814)
INFO 2026-02-02 17:55:25.291572 UTC Successfully collected 3,814 advisories
INFO 2026-02-02 17:55:25.291680 UTC Step [collect_and_store_advisories] completed in 42 seconds
INFO 2026-02-02 17:55:25.291715 UTC Step [clean_downloads] starting
INFO 2026-02-02 17:55:25.291739 UTC Removing cloned repository
INFO 2026-02-02 17:55:25.328524 UTC Step [clean_downloads] completed in 0 seconds
INFO 2026-02-02 17:55:25.328685 UTC Pipeline completed in 47 seconds

Process finished with exit code 0
from vulnerabilities.models import AdvisoryV2
from django.db.models import Count
duplicates = (
    AdvisoryV2.objects
    .values('avid')
    .annotate(count=Count('id'))
    .filter(count__gt=1)
)
len(duplicates)
Out[2]: 0
AdvisoryV2.objects.count()
Out[3]: 3814

gentoo importer V1 logs:

Importing data using vulnerabilities.importers.gentoo.GentooImporter
Invalid safe_version 3.24.48:3 - error: '3.24.48:3' is not a valid <class 'univers.versions.GentooVersion'>
Invalid safe_version 6.9.3:6 - error: '6.9.3:6' is not a valid <class 'univers.versions.GentooVersion'>
Successfully imported data using vulnerabilities.importers.gentoo.GentooImporter

@ziadhany
Copy link
Collaborator Author

ziadhany commented Jan 1, 2026

Signed-off-by: ziad hany <ziadhany2016@gmail.com>
Signed-off-by: ziad hany <ziadhany2016@gmail.com>
Update the Gentoo get_safe_and_affected_versions function in advisory v2

Signed-off-by: ziad hany <ziadhany2016@gmail.com>
Signed-off-by: ziad hany <ziadhany2016@gmail.com>
Signed-off-by: ziad hany <ziadhany2016@gmail.com>
Signed-off-by: ziad hany <ziadhany2016@gmail.com>
Signed-off-by: ziad hany <ziadhany2016@gmail.com>
Signed-off-by: ziad hany <ziadhany2016@gmail.com>
@TG1999 TG1999 requested a review from keshav-space February 4, 2026 12:06
Copy link
Member

@keshav-space keshav-space left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ziadhany Thanks, see some feedback below.

cves = []
summary = ""
xml_root = ET.parse(file).getroot()
id = xml_root.attrib.get("id")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
id = xml_root.attrib.get("id")
id = xml_root.attrib.get("id", "")


affected_package = AffectedPackageV2(
package=purl,
affected_version_range=EbuildVersionRange(constraints=[constraint]),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It does not make sense to report unaffected range in affected_version_range.

Comment on lines +111 to +113
url=f"https://security.gentoo.org/glsa/{id}"
if id
else "https://security.gentoo.org/glsa",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should suffice

Suggested change
url=f"https://security.gentoo.org/glsa/{id}"
if id
else "https://security.gentoo.org/glsa",
url=f"https://security.gentoo.org/glsa/{id}",

Comment on lines +144 to +145
if invert:
constraint = constraint.invert()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ziadhany why do we invert fixed range, we should report fixed range as is.

Comment on lines +37 to +50
{
"package": {
"type": "ebuild",
"namespace": "dev-vcs",
"name": "subversion",
"version": "",
"qualifiers": "",
"subpath": ""
},
"affected_version_range": "vers:ebuild/<=1.8.18",
"fixed_version_range": null,
"introduced_by_commit_patches": [],
"fixed_by_commit_patches": []
}
Copy link
Member

@keshav-space keshav-space Feb 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not right, for the unaffected range <unaffected range="ge">1.9.7</unaffected> and <unaffected range="rgt">1.8.18</unaffected> we should report vers:ebuild/>1.8.18 and vers:ebuild/1.9.7 as fixed version range and do not invert these and report it as affected vers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants